Add metric to represent a consumer group's total offset lag per topic #93
Conversation
|
Hi @dylanmei, Thank you for your contribution! We really value the time you've taken to put this together. Before we proceed with reviewing this pull request, please sign the Lightbend Contributors License Agreement: |
There was a problem hiding this comment.
Thanks for the PR. I see the code is grouping by consumergroup topic to take the sum. In cases where a consumer group is subscribed to more than 1 topic that means you could have multiple total lag metrics per group. I can see how it would make sense to report total lag for a particular group topic, but it would also be logical to sum lag for all partitions across all topics. WDYT?
src/test/scala/com/lightbend/kafkalagexporter/ConsumerGroupCollectorSpec.scala
Outdated
Show resolved
Hide resolved
My initial motivation was to provide a low-cardinality roll-up of the partition lag, especially as values such as At some point aggregations are better handled at query time, and I think "logical to sum lag for all partitions across all topics" may be one such case. However, it's non-trivial to add in if today if you feel strongly about it. |
I think it would still satisfy your low-cardinality use case. It would be a more simple implementation as well because it would follow the same logic as the Ex) My consumer group has a subscription of |
src/main/scala/com/lightbend/kafkalagexporter/ConsumerGroupCollector.scala
Show resolved
Hide resolved
dylanmei
force-pushed
the
consumer_group_lag_by_topic
branch
2 times, most recently
from
1c5436e to
f233634
Compare
|
Updating after initial feedback. New metrics:
Labels: The sum of the difference between the last produced offset and the last consumed offset of all partitions for this group.
Labels: The sum of the difference between the last produced offset and the last consumed offset of all partitions in this topic for this group. I'm still insecure about the name of this second metric: To follow the "including the aggregation type in the metric name" guidance @seglo has given, it could also be |
Looking good @dylanmei ! Let's call it |
dylanmei
force-pushed
the
consumer_group_lag_by_topic
branch
from
f233634 to
279b273
Compare
…seglo#93) * Add metric of a consumer group's total offset lag and total lag per topic * Update name of group/topic aggregate metric
In regards to #92, add a metric to represent a consumer group's total offset lag per topic.
New metric:
kafka_consumergroup_group_topic_lagLabels:
cluster_name, group, topicThe sum of the difference between the last produced offset and the last consumed offset of all partitions in this topic for this group.
I'm on fence about the metric name itself. Some things that came to mind:
kafka_consumergroup_group_topic_lag(current proposal)kafka_consumergroup_group_total_lag(original proposal)kafka_consumergroup_topic_lag